1. Technical Field
The invention relates to a method for representing virtual information in a view of a real environment comprising the steps of providing a system setup with at least one display device, wherein the system setup is adapted for blending in virtual information on the display device. The invention also relates to a computer program product comprising software code sections for performing the method.
2. Background Information
Augmented reality (AR) systems are known to enhance information of a real environment by providing a visualization of overlaying computer-generated virtual information with a view of the real environment or a part of the real environment. The virtual information can be any type of visually perceivable data such as objects, texts, drawings, videos, or their combination. The view of the real environment or the part of the real environment, as understood herein, could be perceived as visual impressions by user's eyes and/or be acquired as one or more images by a camera, e.g., worn by a user or attached on a device held by a user.
The overlaid or blended in virtual information may be, in principle, various items of virtual information. For example, an item of virtual information which could enhance information of a real environment may be a point of interest, as for example known in map or navigation applications. A point of interest (POI) may represent a location of a real object of the real environment (e.g., a building or a landmark) and often includes digital content that is related to the real object. For instance, the location is a global location (e.g., a geo-coordinate such as a 2D coordinate of longitude and latitude, or a 3D coordinate of longitude, latitude and altitude) or a postaddress (e.g., a floor number, street, postcode, country). The post address and the global location could be converted to each other. The digital content of the POI could contain various data, such as a name, description, and contact related to the real object.
One major function of augmented reality systems is to overlay items of virtual information, such as points of interest (POIs), to a view of the real environment. This is particularly useful and popular in location-based (mobile) augmented reality applications, such as tour guidance for exploring the urban environment, as described in references [1, 2, 4] as referred to herein at the end of the description. For example, users could use an augmented reality application to overlay POIs to a view of the real environment when they are in new places and want to obtain information about things they see. In augmented realty systems, the POI information has to be represented in the real environment or the part of the real environment such that it satisfies desired visual perception and usability. Most data sources for POI information provide the information in relation to a point in the real world, not as a 3d model with more than one point or vertex. This invention is well suited for handling the visualization of items of virtual information, which are described by latitude, longitude and optionally altitude and additional meta-information consisting of text and 2D image information.
Several methods have been developed for improving visual perception and usability of blending in or overlaying POI information to a view of a real environment in augmented reality applications.
View distance of a POI refers to a distance between the location of the POI and the location where a camera or a user stands. View direction of a POI refers to a direction from the location where a camera or a user stands to the location of the POI.
Hoellerer et al. in reference [4] propose to display POIs as virtual flags and labels in the correct perspective for the user's view pose, while the labels face the user and maintain their size irrespective of distance to ensure readability.
Augmented reality systems face significant technical challenges to more usefully display POIs to users. Uusitalo et al. in reference [1] disclose a method for displaying POI information based on partitioning of the real environment. The method determines to overlay one or more POIs based on the one or more partitions of the view of the real environment. They teach utilizing the knowledge of floor plan or height of a building to separate the building into partitions and overlaying the POIs to corresponding partitions in an image of the buildings.
When POIs have the same view direction, the POIs may be arranged behind with each other for display in augmented reality applications. In this regard, some of the POIs may not be visible, as it may be hidden behind other POIs. For this, Sandberg in reference [2] motivates grouping POIs that have the same view direction and then displaying the grouped POIs in a visible manner.
Meier et al. in reference [3] disclose a method to achieving an ergonomic representation of POIs in augmented reality systems, by subdividing the view of the real environment into a plurality of regions based on the distances of the regions to the view point.
Augmented reality systems commonly overlay the POIs opaquely on the top of a view of the real environment. An exemplary scenery is shown in FIG. 9, where POIs are overlaid as opaque labels 92, 93, 94, 95, 96, 97, 98, and 99 on the top of an image 81 of a real environment 51. In this way, the POI representations (e.g. the opaque labels) occlude the real environment or the part of the real environment in the image. Sandberg in reference [2] and Uusitalo et al. in reference [1] also overlay the icons and labels of the POIs opaquely to an image of a real environment. This introduces difficulties for the users to quickly and intuitively understand the real environment from the overlaid POIs.
Real objects of the real environment sometimes may not be visible in the view of the real environment, as the objects could be occluded by real objects placed in front of them from the view point. An example is shown in FIG. 8 (showing the scene of FIG. 9 without POIs), where a real object (building) D (cf. FIG. 7) in the real environment 51 is not visible in the image 81 of the environment. In this situation, simply superimposing the POIs related to the occluded objects on the top of the view of the real environment (as shown in FIG. 9) could confuse users in a way that the users may wrongly relate the POIs to a front object which is visible in the view. In the example of FIG. 9, the users may relate the POI information 93, 94, 95, 96, 97, and 98 to a real object (building) A, while 97 and 98 that represent POI_7 and POI_8 are not related to the building A.
Therefore, it would be desirable to visualize the POIs related to the occluded objects in a different way from the visible front objects and more particularly in a way that the users could intuitively perceive the occlusion effect. For example, the POIs related to the occluded objects may be shown semi-transparent, or in dash lines, while the POIs related to the front objects could be shown opaquely, in solid lines, or in a different color.
Meier et al. in reference [3] teach to use the depth along the view direction of a POI from the view point to the real environment in order to determine whether to display the POI in an occlusion model or not. If the POI has a longer view distance than the depth, the POI is displayed in an occlusion way, i.e. semi-transparent, or in dash lines, according to reference [3].